Actionable Auditing Revisited

Although algorithmic auditing has emerged as a key strategy to expose systematic biases embedded in software platforms, we struggle understand the real-world impact of these audits and continue find it difficult translate such independent assessments into meaningful corporate accountability. To analyze publicly naming disclosing performance results biased AI systems, investigate commercial Gender Shades, first audit gender- skin-type disparities facial analysis models. This paper (1) outlines design structured disclosure procedure used Shades study, (2) presents new metrics from targeted companies IBM, Microsoft, Megvii (Face++) on Pilot Parliaments Benchmark (PPB) August 2018, (3) provides PPB by non-target Amazon Kairos, (4) explores differences company responses shared through communications that contextualize PPB. Within 7 months original audit, all three targets released application program interface (API) versions. All reduced accuracy between males females darker- lighter-skinned subgroups, with most significant update occurring for darker-skinned female subgroup underwent 17.7--30.4% reduction error periods. Minimizing led 5.72--8.3% overall target corporation APIs. The non-targets Kairos lags significantly behind targets, rates 8.66% 6.60% overall, 31.37% 22.50% darker subgroup, respectively. is an expanded version earlier publication results, revised more general audience, updated include commentary further developments.